Add instrumentation for chat.completions.parse() (structured outputs)#18
Open
Nik-Reddy wants to merge 1 commit into
Open
Add instrumentation for chat.completions.parse() (structured outputs)#18Nik-Reddy wants to merge 1 commit into
Nik-Reddy wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds OpenAI structured outputs telemetry coverage by wrapping chat.completions.parse() and handling Pydantic model response_format values in request attribute extraction.
Changes:
- Adds parse wrapping/unwrapping for sync and async chat completions when supported.
- Maps Python type response formats to JSON/JSON schema telemetry attributes.
- Adds sync/async structured-output tests and VCR cassettes for content/no-content modes.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
instrumentation/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/__init__.py |
Adds parse support detection and wrappers. |
instrumentation/opentelemetry-instrumentation-openai-v2/src/opentelemetry/instrumentation/openai_v2/utils.py |
Handles type-based response_format values. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/test_structured_outputs.py |
Adds sync structured-output tests. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/test_async_structured_outputs.py |
Adds async structured-output tests. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/structured_outputs_utils.py |
Adds shared structured-output test model and prompts. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_structured_output_with_content.yaml |
Adds sync content-mode cassette. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_structured_output_no_content.yaml |
Adds sync no-content cassette. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_async_structured_output_with_content.yaml |
Adds async content-mode cassette. |
instrumentation/opentelemetry-instrumentation-openai-v2/tests/cassettes/test_async_structured_output_no_content.yaml |
Adds async no-content cassette. |
Comment on lines
+204
to
+208
| self._parse_supported = _is_parse_supported() | ||
| if self._parse_supported: | ||
| wrap_function_wrapper( | ||
| "openai.resources.chat.completions", | ||
| "Completions.parse", |
Comment on lines
+33
to
+34
| def test_structured_output_with_content( | ||
| span_exporter, log_exporter, openai_client, instrument_with_content, vcr |
Comment on lines
+33
to
+34
| @pytest.mark.asyncio() | ||
| async def test_async_structured_output_with_content( |
Wrap Completions.parse and AsyncCompletions.parse with the same telemetry wrappers used for create(). ParsedChatCompletion extends ChatCompletion so the existing wrapper logic handles it correctly. A version guard (_is_parse_supported) skips wrapping on openai < 1.40.0 where parse() does not exist. response_format handling in both the legacy and experimental paths now recognises a bare Python type (Pydantic model class) and maps it to the appropriate output type attribute (json on experimental, json_schema on legacy). Includes sync and async tests with VCR cassettes, skipped on older SDK versions.
Nik-Reddy
force-pushed
the
feat/structured-outputs
branch
from
8e8b16b to
489758a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12
Background
Originally tracked in opentelemetry-python-contrib#3449 with a previous implementation in opentelemetry-python-contrib#4416, which was approved by @lmolkova and @lzchen. That PR was closed when the GenAI instrumentations moved to this repository. This PR ports the same approach to the new repo, aligned with its current structure.
Changes
Adds telemetry coverage for OpenAI's
chat.completions.parse()method, which powers structured outputs. Theparse()method was introduced in openai SDK 1.40.0 and returns aParsedChatCompletionthat extendsChatCompletionwith a typed.parsedfield on each choice message.Instrumentation (
__init__.py)_is_parse_supported()version guard that checks forhasattr(Completions, "parse"). On openai < 1.40.0 the wrapping is skipped entirely (the repo pins openai >= 1.26.0).Completions.parseandAsyncCompletions.parseare wrapped with the same telemetry functions used forcreate(). SinceParsedChatCompletioninherits fromChatCompletion, the existing wrapper logic (span attributes, metrics, content capture) handles it without changes.unwrap()calls added to_uninstrument().response_format type handling (
utils.py)When
parse()is called withresponse_format=SomePydanticModel, the SDK sends a bare Python type rather than a dict or string. Both attribute extraction paths now handle this:get_llm_request_attributes):isinstance(response_format, type)branch setsgen_ai.openai.request.response_format = "json_schema"via theGenAiOpenaiRequestResponseFormatValues.JSON_SCHEMAsemconv constant.create_chat_invocation): same check setsgen_ai.output.type = "json"via theGenAiOutputTypeValues.JSONsemconv constant.Tests
test_structured_outputs.py/test_async_structured_outputs.py: sync and async tests covering both content-capture and no-content modes.response.choices[0].message.parsed(the structured output field).gen_ai.output.type/gen_ai.openai.request.response_formatattribute based on semconv mode.pytestmark.CHANGELOG
Adds an entry under
## Unreleasedreferencing this PR.Acceptance criteria (from #12)
create()callsgen_ai.output.typeis set tojsonwhenresponse_formatis a Pydantic model classcreate()callsWhat's not changed
parse()are non-streaming only.